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INTRODUCTION 


The  Institute  for  Human  and  Machine  Cognition  (IHMC)  is  pleased  to  submit  a  report  of 
progress  on  the  continuation  project  Human  Systems  Technology  for  the  2004  fiscal  year. 
This  fiscal  year  work  actually  began  on  April  1 ,  2004  and  was  completed  on  September 
30,  2005.  Also  called  Human-Centered  Computing,  this  multidisciplinary  field  exploits 
advances  in  cognitive  research  together  with  those  in  computer  science  and  related  areas 
to  optimize  the  cognitive,  perceptual,  and/or  physical  performance  of  experts  and  expert 
teams  and  the  information  systems  that  support  them.  The  work  reported  on  extends  that 
conducted  in  the  second  increment  of  ONR  funding  for  Human  Systems  Technology.  The 
FY  2004  work  focused  in  particular  on  (1)  advanced  visual  information  displays 
exploiting  the  effectiveness  with  which  the  human  brain  processes  spatial  orientation 
information;  (2)  advanced  tactile  displays  that  provide  enhanced  situational  awareness  in 
such  complex  operational  domains  as  aviation  and  special  forces  operations;  and  (3) 
continued  development  of  improved  algorithms  for  Knowledge  Discovery  and  Data 
Mining  (KDD)  from  large  data  sets  and  associated  investigations  of  displays  and  training 
principles  to  improve  human  abilities  to  rapidly  diagnose  failures  in  complex  systems. 
The  work  in  each  of  these  areas  reflects  the  growing  appreciation  for  the  enormous 
potential  that  information  technology  has  to  leverage  and  amplify  human  capabilities. 
Realization  of  this  potential  requires  a  deep  understanding  of  human  cognition, 
perception,  and/or  locomotion;  the  relevant  areas  of  computer  science;  and  the  nature  of 
the  human  activity  to  be  enabled.  This  inherently  multidisciplinary  approach  to  realizing 
the  foil  potential  of  emerging  information  technology  capabilities  is  at  the  heart  of  the 
Institute  for  Human  and  Machine  Cognition  (IHMC)  and  the  research  conducted  during 
FY  2004. 

In  what  follows  the  progress  made  in  each  of  these  areas  is  described,  along  with  a  brief 
statement  of  plans  for  FY  05.  This  includes  a  list  of  publications  resulting  from  the  work. 
During  the  performance  period  a  total  of  16  publications  have  been  prepared  and 
published,  are  in  press,  are  currently  under  review. 

OZ:  A  HUMAN-CENTERED  COMPUTING  COCKPIT  DISPLAY 

Background 

Contemporary  instrument  design  for  aircraft  is  inherently  inefficient  in  that  it  forces  the 
pilot  to  view  each  instrument  sequentially  in  order  to  gather  information.  This  process, 
called  the  'instrument  scan,'  is  a  huge  visual  and  mental  workload  for  the  pilot  engaged  in 
instrument  flight,  and  this  presentation  format  does  not  facilitate  integration  of  the 
information  by  the  pilot  to  identify  flight  conditions  that  may  be  dangerous.  Drs.  David 
Still  (Institute  for  Human  and  Machine  Cognition)  and  Leonard  Temme  (Naval 
Aerospace  Medical  Research  Laboratory)  have  designed  an  alternative  method  of 
presenting  cockpit  information,  OZ,  which  capitalizes  on  what  the  human  eye  was 
designed  to  see  best,  quickest,  and  easiest.  It  is  an  integrated  display  of  all  relevant 
information  in  a  manner  that  is  both  intuitive  and  takes  foil  advantage  of  the  bandwidth 
of  both  peripheral  and  fovial  vision.  OZ  allows  pilots  to  view  an  integrated  picture  of 


how  different  flight  parameters  affect  aircraft  performance  instead  of  having  to  mentally 
input  individual  pieces  of  information  into  a  memorized  flight  model  and  then  generate 
performance  predictions.  Furthermore,  OZ  symbology  can  tolerate  substantial  visual 
insult  in  the  form  of  blur  and  laser  trauma.  IHMC  and  NAMRL  evaluations  of  OZ  have 
found  that  non-pilots  learned  flight  tasks  faster  and  performed  better  with  OZ  than  with 
conventional  instruments.  Additionally,  for  both  rated  pilots  and  non-pilots,  as  flight 
tasks  become  more  challenging,  performance  deteriorated  significantly  with  conventional 
displays  but  remained  unaffected  with  OZ. 

OZ  consists  of  abstract  metaphorical  objects  that  convey  multiple  data  streams  of  aircraft 
state  parameters  by  the  scale  of  the  objects’  component  parts  and  that  indicate  the 
underlying  controlling  algorithms  by  the  objects’  configurations. 

•To  reduce  time  restrictions  imposed  by  sequential  viewing,  the  metaphorical 
objects  are  constructed  such  that  their  informational  content  can  be  acquired 
without  the  use  of  central  vision. 

•To  increase  information  density  and  decrease  visual  clutter,  OZ  relies  upon  such 
visual  perception  concepts  as  structure- from-motion,  pop  out,  texture,  continuity, 
and  figure-ground  discrimination  to  construct  its  metaphorical  objects. 

•To  reduce  mental  computation  requirements  of  the  pilot,  reference  values,  data 
streams,  and  performance  capabilities  are  adjusted  for  current  aircraft 
configuration  and  environmental  conditions. 

•To  improve  situational  awareness,  OZ  maps  external  space  with  the  same  wide- 
angle  frame  of  reference  used  to  show  aircraft  orientation,  location,  and 
performance. 

•To  improve  the  pilot’s  ability  to  simultaneously  perform  secondary  tasks,  OZ 
allows  the  pilot  to  fly  with  peripheral  vision  while  using  central  vision  to  read 
checklists  or  inspect  secondary  displays. 

•To  aid  target  acquisition  and  reacquisition  sensor  images  (infrared,  image 
intensification,  and  radar),  both  the  object  of  regard  and  the  sensor’s  field  of  view 
can  be  spatially  located  within  the  OZ  frame  of  reference. 

The  overall  result  is  that  OZ  shifts  the  workload  requirements  for  flight  from  one  of 
visual  scanning  of  separate  instruments  and  displays,  requiring  intensive 
integration/computation,  to  nearly  instantaneous  or  ‘direct’  perception  of  an  integrated 
picture.  In  essence,  a  glance  at  OZ  lasting  200  ms  conveys  most  of  the  information 
contained  in  the  entire  panel  of  conventional  instruments  that  may  take  several  seconds  to 
scan.  Although  OZ  presents  the  pilot  with  processed  flight  data,  the  processing  does  not 
obscure  information  nor  does  it  make  decisions  for  the  operator. 

According  to  the  National  Academy  of  Science,  the  Navy  will  need  to  develop  the 
appropriate  man-machine  interface  for  the  UCAV-N  because  the  civilian  sector  is 
unlikely  to  provide  them  (NAS  Review  of  ONR’s  Uninhabited  Combat  Air  Vehicles 
Program,  2000).  The  continued  incremental  development  of  conventional 
instrumentation  is  unlikely  to  provide  the  required  interface.  The  solution  we  propose, 
OZ,  uses  the  design  freedom  allowed  by  new  computer  technology  to  take  advantage  of 


recent  discoveries  in  vision  and  cognitive  sciences  to  make  a  display  that  is  tailored  to  the 
biology  of  the  observer.  The  end  result  is  a  powerful  cockpit  environment  that  meets  the 
needs  of  the  UCAV-N  community. 

Progress  to  Date 

During  the  course  of  this  years  funding,  IHMC  produced  a  design  for  the  OZ  UCAV 
Special  Warfare  display  and  implemented  several  of  its  key  components. 

1.  Translation  and  motion  cues  were  implemented  that  enable  the  starfleld  to  move 
as  a  mirror  image  of  the  aircrafts  movement.  This  gives  an  intuitive  notion  of 
slight  changes  in  elevation  or  orientation,  and  also  dramatically  increases  operator 
sensitivity  to  fore-aft  and  lateral  drift. 

2.  Additional  waypoints  and  ground  reference  markers  were  added.  The  altitude, 
size,  and  latitude/longitude  location  of  waypoints  in  the  sky  can  be  read  from  a 
data  file  and  displayed  in  OZ  (when  inside  the  maximum  distance  bound).  These 
waypoints  have  been  tested  as  part  of  the  integration  with  the  Predator  STE 
simulation  software.  Using  waypoints  to  mark  critical  decision/evaluation  points 
in  training  tasks  (such  as  landing)  significantly  reduce  the  mental  effort  required 
to  complete  the  training  tasks,  and  greatly  ramps  performance  to  test  criteria.  New 
ground  markers  were  added  to  denote  points  of  interest.  They  are  created  by  the 
operator  while  in  flight  and  can  be  shown  or  not  based  on  a  distance  filter.  The 
ground  markers  referencing  GPS  coordinates  of  the  runway  centerline  were 
refined  and  tested  in  the  context  of  severe  crosswind  landings.  The  extended 
runway  centerline  has  demonstrated  effectiveness  in  crosswind  landings  and  for 
general  runway  alignment  issues. 

3.  Wind  indications  were  implemented  that  intuitively  demonstrate  the  effects  of  the 
wind  on  the  ground-track  of  the  aircraft.  By  integrating  a  wind  direction 
reference  into  the  nose-ring  of  the  OZ  flight  model,  the  operator  is  continually 
reminded  of  the  wind  direction.  The  effects  of  the  wind  are  also  indicated  by  the 
offset  of  the  nose-ring  from  centerline.  This  integration  allows  both  no  crosswind 
and  crosswind  landings  to  be  treated  the  same:  as  long  as  the  airport  runway  is 
depicted  inside  of  the  nose-ring,  the  aircraft  is  headed  directly  for  the  runway. 
This  eliminates  special  calculations/training  for  determining  crosswind  drifts,  as 
this  is  computed  and  directly  displayed  in  OZ. 

4.  An  option  to  de-clutter  the  interface  was  implemented,  where  the  background 
starfield  can  disappear  if  the  operator  is  maintaining  the  selected  altitude  and 
heading.  When  the  operator  deviates  by  more  that  100  ft  in  altitude  or  10  degrees 
in  heading,  the  starfield  reappears.  It  is  important  to  distinguish  this  from  a 
simple  alert  sound,  or  a  flashing  message  on  the  screen:  This  method  of 
decluttering  depends  on  operator  adherence  to  a  specified  plan  (i.e.,  heading, 
altitude).  It  makes  the  interface  “active”  in  the  sense  that  it  now  unobtrusively 
gets  the  operators  attention  by  displaying  the  exact  information  that  is  needed  to 
correct  the  deviation,  rather  than  by  producing  a  sound  or  message  that  then 
requires  interpretation  to  be  understood. 


A  multi-user  capability  was  implemented  that  allows  for  communication  of  speed, 
altitude,  and  heading  information  to  be  transmitted  between  separate  OZ  installations. 
This  permits  other  aircraft  in  a  group  to  be  visually  located  within  OZ  and  tracked. 

Proposed  Work 

This  activity  has  now  reached  a  high  degree  of  maturity,  so  FY  2005  activities  will  be 
limited  to  facilitating  transition  to  application. 

PSYCHOPHYSICAL  STUDIES  OF  TACTILE  INTERFACES  FOR  SENSORY 
SUBSTITUTION  OR  AUGMENTATION  IN  COMPLEX  TASKS 

Background 

The  US  Navy  has  an  existing  requirement  for  tactile  interfaces  to  improve  situation 
awareness.  Tactile  interfaces  have  been  developed  for  sensory  substitution  applications 
such  as  aids  for  the  vision  or  hearing  impaired.  Tactile  interfaces  have  also  been 
developed  for  sensory  augmentation  in  complex  dynamic  environments  such  as  aviation 
applications.  In  the  former  application  great  effort  is  taken  to  represent  the  sensory 
information  in  as  high  resolution  as  possible  (e.g.,  to  let  the  operator  “see”  detail  with  the 
sense  of  touch.)  The  latter  application  seeks  to  reduce  the  resolution  to  the  most  base 
useful  representation  of  particular  sensor  information  in  order  to  reduce  the  cognitive 
workload  of  understanding  the  information  displayed  tactually.  In  their  current 
incarnations,  both  methods  operate  without  competition  for  sensory  or  cognitive  assets, 
as  little  other  tactile  information  is  useful  in  their  respective  operational  environments. 

In  1995,  the  Naval  Aerospace  Medical  Research  Laboratory  (NAMRL)  and  the 
University  of  West  Florida  (UWF)  Institute  for  Human  and  Machine  Cognition  (IHMC) 
began  collaborating  on  flight-testing  NAMRL’s  Tactile  Situation  Awareness  System 
(TSAS).  The  relationship  has  continued  to  develop  and  conduct  demonstrations  with  a 
suit  that  provides  pilots  with  spatial  orientation  information  via  tactile  stimulation  in 
order  to  reduce  the  number  of  SD  accidents  occurring  with  pilots.  The  suit  displayed 
ground  reference  information  from  the  attitude  indicator  to  orient  blind  folded  pilots.  The 
suit  was  successfully  flight  tested  on  a  blind  folded  pilot  of  a  fixed-wing  (T-34)  aircraft 
and  on  several  blind  folded  pilots  of  a  rotary-wing  (UH-60)  aircraft.  The  UH-60  pilots 
successfully  flew  basic  instrument  tasks  including  unusual  attitude  recovery  and  in  the  T- 
34,  the  pilot  successfully  flew  acrobatic  flight  maneuvers  in  addition  to  instrument  flight 
tasks. 

In  September  1997,  NAMRL  and  IHMC  developed  an  improved  air-driven  tactile  display 
to  provide  Differential  Global  Positioning  Satellite  (DGPS)  based  drift  information  to 
UH-60  pilots  for  the  Office  of  Joint  Strike  Fighter  (JSF).  Seven  pilots  used  the  tactile 
display  vest  and  hovered  the  UH-60  with  limited  outside  visual  cues  (their  vision  was 
obscured  to  20/200  by  optical  filters).  This  demonstration  was  performed  to  show  that 
pilots  could  obtain  drift  and  rate  of  drift  information  through  the  tactile  modality 
sufficiently  to  hover.  A  number  of  flight  maneuvers  were  performed,  including  stationary 


hover  in  and  out  of  ground  effect  (IGE,  OGE),  translational  maneuvers  and  simulated 
shipboard  take-offs  and  landings. 

Further  rotary  wing  and  unmanned  aerial  vehicle  (UAV)  flight-testing  and  laboratory 
simulations  have  further  demonstrated  the  effectiveness  of  tactile  interfaces  for  situation 
awareness.  TSAS  like  any  other  avionics  system  is  susceptible  to  feature  creep  as  more 
end  users  evaluate  and  understand  the  benefits.  In  order  to  provide  more  information 
tactually,  the  data  must  either  be  intelligently  scheduled  for  display  or  presented  at  higher 
resolution.  The  former  control  methodology  is  currently  under  investigation  by  this 
research  team.  The  latter  will  be  investigated  under  this  project. 

High-resolution  tactile  displays  can  only  be  effective  if  the  areas  they  contact  are  highly 
innervated.  Mapping  of  sensitivities  has  shown  that  the  highest  resolution  of  innervation 
occurs  in  the  hands,  face  and  tongue.  In  the  aviation  environment  the  tongue  makes  for  an 
ideal  location  for  a  high-resolution  tactile  interface.  Unlike  the  hands  it  is  not  in  active 
use,  except  during  verbal  communications,  and  it  has  no  keratinized  skin  layer,  allowing 
for  more  efficient  transduction  of  the  tactile  signal.  We  will  use  our  tongue  based  tactile 
interface  to  provide  higher  resolution  information  in  concert  with  the  lower  resolution 
information  of  TSAS.  This  will  represent  a  confluence  of  two  tracks  of  thought  on  tactile 
displays,  one  of  providing  sensory  augmentation  displaying  intuitive,  dimensional/ 
directional  information  (as  with  TSAS)  and  the  other  of  sensory  substitution  where 
information  normally  perceived  via  the  eyes  or  ears  is  perceived  through  a  tactile 
interface.  The  goal  of  this  integration  will  be  to  create  a  tactile  interface  that  operates,  in 
3D,  in  a  manner  similar  to  the  eye,  with  low  resolution  “peripheral”  sensation  provided 
on  the  torso  by  TSAS  and  high  resolution  “foveal”  sensation  provided  on  the  tongue.  The 
resultant  multi-faceted  tactile  display  will  be  tested  in  concert  with  visual  and  audio 
displays  using  a  fixed  base  simulator  at  IHMC. 

Progress  to  Date 

We  have  integrated  our  Adaptive  Multiagent  Integration  (AMI)  architecture,  developed 
under  funding  from  DARPA  with  our  tactile  interface  devices  (the  TSAS  torso  tactile 
interface  and  BrainPort™)  to  facilitate  integrated  testing.  We  are  awaiting  delivery  of  a 
current  generation  version  of  the  Wicab  BrainPort™  to  use  in  our  FY05  integrated  high 
and  low  resolution  tactile  display  protocol.  Custom  torso  tactile  interface  hardware 
fabricated  in  FY04  has  been  redesigned  and  revised  to  improve  discrete  control  of  the 
torso  tactile  interface  and  to  enable  use  of  intelligent  automation  features  made  available 
through  integration  with  AMI.  Two  journal  articles  have  been  drafted  describing  the 
results  of  the  FY04  experiments  (presented  at  the  Aerospace  Medical  Association  2004 
Annual  Scientific  Meeting,  abstract  numbers  330-Tactile  Display  landing  Safety, 
Situational  Awareness,  and  Workload  Reduction  Improvements  for  the  Space  Shuttle, 
and  331 -An  Application  of  the  Tactile  Situation  Awareness  System  to  the  Learning  of  a 
Complex  Dynamic  Task  -  ILS  Approach  and  Landing).  One  article  is  being  revised 
following  initial  peer  review,  the  second  is  in  final  draft  for  initial  submission  (estimated 
date  of  submission  is  June  05,  following  completion  of  post  hoc  analysis  and  full  analysis 
of  pilot  studies. 


The  pilot  studies  included  a  dynamic  tracking  task  based  on  the  Space  Shuttle  Head  Up 
Display  (HUD)  approach  and  landing  deviation  display.  In  the  first  condition,  the 
guidance  and  velocity  vector  icons  were  driven  in  a  pseudorandom  fashion  using  the  sum 
of  5  sinusoidal  waveforms  (0.04,  0.08,  0.12,  0.16  and  0.22  Hz)  in  the  X  and  Y  axes.  The 
subjects  were  instructed  to  use  the  rotational  hand  controller  joystick  to  keep  the  two 
icons  superimposed.  A  second  condition  used  surround  sound  (visual  screens  were  blank) 
to  provided  the  directional  error  between  the  two  data  points  (driven  by  the  same 
sinusoids)  and  a  third  condition  provided  the  subjects  with  the  same  deviation  display 
using  TSAS.  Four  more  conditions  that  represented  combinations  of  visual,  audio  and 
tactile  stimuli  were  presented  in  a  counterbalanced  fashion.  Fourier  analysis  was 
performed  to  determine  the  gain  and  phase  for  each  display  modality  at  the  stimulus 
frequencies  (see  Figure  1). 


Figure  1:  Bode  plots  for  the  seven  pilot  study  conditions.  Gain  of  1  and  phase  lead/lag  of 
0  degrees  are  ideal. 


The  initial  analysis  demonstrates  that  the  tracking  task  was  sufficiently  difficult,  even  at 
relatively  low  frequencies,  to  induce  large  pilot  induced  oscillations  (large  gains  with 
phase  shifts).  Given  that  the  task  was  based  on  a  visual  tracking  paradigm,  it  is  not 
surprising  to  see  that  the  visual  only  presentation  provided  the  best  response  in  gain  (both 
axes).  The  multi-modal  display  (visual+audio+tactile)  is  however  the  next  best  across  the 
majority  of  the  frequencies  tested.  It  is  obvious  from  the  data  that  the  high  frequencies  in 
this  task  were  too  fast  for  accurate  tracking.  Follow  up  studies  will  focus  on  the  range 
from  0.04  to  0.1  Hz  to  better  characterize  where  the  transition  to  over  compensatory 
response  arises.  Additionally,  we  will  investigate  the  tracking  response  to  single 


frequency  sinusoids  to  verify  that  the  pseudorandom  nature  of  the  tracking  task  did  not 
adversely  affect  individual  performance. 

Tables  1-4  provide  the  means  and  standard  deviations  for  the  average  (n=14)  gain  and 
phase  responses  depicted  in  Figure  1 . 


Table  2.  Phase  means  and  standard  deviations  for  X  Position. 
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Table  1.  Gain  means  and  standard  deviations  for  X  Position. 
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Table  3.  Gain  means  and  standard  deviations  for  Y  Position. 
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Table  4.  Phase  means  and  standard  deviations  for  Y  Position. 

The  work  supported  by  ONR  has  also  resulted  in  the  preparation  or  submission  of  the 
following  manuscripts: 


Raj,  A.  K.,  Higgins,  J.,  Abraham,  L.  J.,  Carff,  R.  W.,  &  Rupert,  A.  H.  (in  preparation). 
An  application  of  the  tactile  situation  awareness  system  (TSAS)  to  the  learning  of 
a  complex  dynamic  task  -  ILS  approach  and  landing, 

Olson,  J.  M.,  Raj,  A.  K.,  Thomas,  R.  E.  and  Davis,  J.  A.  (in  preparation  following  initial 
peer  review)  Tactile  Display  Landing  Safety,  Situational  Awareness,  and 


Workload  Reduction  Improvements  for  the  Space  Shuttle  (submitted  to  Aerospace 
and  Environmental  Medicine) 


Proposed  Research 

In  the  work  supported  with  FY  2005  hinds,  we  will  continue  to  use  our  team’s  expertise 
the  two  fields  of  tactile  interface  implementation  (sensory  augmentation  and  sensory 
substitution)  to  evaluate  the  psychophysical  effects  of  combining  both  types  of  tactile 
interfaces  into  a  single  unified  cockpit  interface.  Combining  the  tactile  interfaces  has 
been  enabled  using  the  AMI  architecture  and  will  prove  useful  in  situations  where  the 
high-resolution  component  would  augment  synthetic  vision  and  the  low-resolution 
component  would  augment  spatial  awareness.  Specific  intelligent  control  methods  will 
also  be  developed  to  manage  this  proposed  multi-faceted  tactile  interface  to  minimize 
cognitive  workload.  A  study  will  be  performed  to  evaluate  the  effectiveness  of  the  multi¬ 
faceted  tactile  display  as  an  adjunctive  avionics  display  in  specific  standard  flight 
maneuvers. 

ADVANCED  ALGORITHMS  FOR  HUMAN  SYSTEMS  TECHNOLOGY: 
Knowledge  Discovery  and  Data  Mining  and  Human  Causal  Learning 

Background 

The  rapid  development  of  data  mining  and  machine  learn  algorithms  has  contributed  to 
progress  in  several  areas  of  importance  to  the  Navy: 

1.  Ability  to  recognize  and  classify  from  databases  with  large  numbers  of  variables  and 
comparatively  small  samples  of  messy  data. 

Data  of  this  kind  often  arise  in  intelligence  analysis,  medical  analysis,  automated 
text  interpretation  and  elsewhere. 

2.  Ability  to  learn  causal  mechanisms  from  distributed  datasets  with  distinct  variable  sets. 

Data  are  often  collected  from  distinct  kinds  of  units  with  distinct  record  keeping 
practices  and  must  be  synthesized  into  a  coherent  causal  representation. 

3.  Design  of  Efficient  Fault  Detection  and  Fault  Location  Algorithms. 

Automated  systems  can  aid  in  both  detecting  and  localizing  faults  in  complex 
systems  and  in  designing  workarounds. 

4.  Development  of  instruction  techniques  and  automated  tutorials  for  improving  human 
judgment  of  cause-effect  relationships  and  diagnosing  causal  structure. 

5.  Derivation  of  social  networks  and  command  and  control  structures  from 
communication  data  and  observed  military  activities. 


6.  Development  of  algorithms  for  inferring  causal  structure  in  dynamical,  non-linear 
systems. 

A  good  deal  of  further  research,  development  and  testing  is  needed  in  several  of  these 
domains. 

Progress  to  Date 

The  structure  of  a  complex  system  consisting  of  many  interconnected  components  may 
be  completely  known,  or  largely  unknown,  or  in  the  case  of  a  complex  system  (an  engine, 
a  power  plant,  etc.)  that  is  behaving  anomalously,  we  may  not  know  what  we  do  or  do  not 
know  about  it.  In  analyzing  an  incompletely  known  system,  or  in  locating  the  sources  of 
faults  in  an  improperly  working  system,  we  would  in  many  cases  like  to  make  maximal 
use  of  data,  and  minimize  the  number  of  interventions  or  probes  required  to  identify  the 
structure  uniquely. 

For  systems  that  are  not  perfectly  deterministic,  there  are  reasonably  fast  computable 
procedures  that  identify  some  of  the  dependency  relations  among  components  from  data 
on  component  behavior.  These  methods  do  not  require  experimental  interventions  or 
probes  than  manipulate  or  control  internal  components  of  the  system;  they  merely  require 
records  of  data  on  component  behavior  for  a  range  of  inputs.  These  methods  have  several 
limitations:  they  do  not  work  well  when  components  are  densely  connected;  they  often 
cannot  determine  the  direction  of  signal  or  influence  between  components;  and  they  do 
not  work  well  for  deterministic  systems. 

Some  of  our  work  under  the  ONR  contract  under  FY  04  funds  has  focused  on  how  to 
merge  these  methods  with  experimental  interventions  so  as  to  minimize  the  number  of 
experimental  interventions  required  to  determine  the  structure  of  the  system. 
Experimental  probes  of  a  system  with  unknown  properties  can  be  risky,  and  data 
collection  is  expensive,  so  minimization  is  important.  Superficially,  the  number  of 
experiments  required  to  determine  all  causal  connections  among  N  components  is  an 
exponential  function  of  N.  For  to  determine  that  a  variable  X  does  not  in  any  state  of  the 
system  directly  influence  another  variable  Y,  one  would,  it  seems,  need  to  consider  every 
set  of  values  of  the  remaining  variables  and  show  that  with  that  assignment  of  values  to 
the  other  N  -  2  variables,  X  and  Y  do  not  covary  when  X  is  randomized. 

To  the  contrary,  we  have  shown  that,  quite  generally,  when  there  are  N  components,  for 
all  of  which  there  are  recorded  values,  N  -1  experiments  that  randomize  one  variable 
(component)  at  a  time  are  always  sufficient  and  in  the  worst  case  that  many  experiments 
are  necessary.  It  may  in  some  cases  be  cheaper  (require  that  fewer  data  cases  be 
collected)  to  randomize  several  variables  simultaneously.  When  one  experiment  can 
randomize  any  number  of  variables,  the  number  of  required  experiments  is  reduced  still 
further,  quite  dramatically  to  log2(N)  +  1.  The  proofs  provide  methods  for  obtaining 
these  optima. 


Our  work  during  the  FY  04  funding  cycle  has  also  progressed  in  the  following  additional 
areas: 

•  Interactive  training  and  instruction  in  causal  inference  (Scheines,  Easterday  and 
Danks),  an  area  we  hope  to  develop  much  further,  eventually  to  yield  training 
methods  for  diagnosing  causes  of  system  failures; 

•  The  psychology  of  human  causal  judgment  (Danks,  Danks  and  Schwartz),  work 
which  helps  to  provide  scientific  foundations  for  the  previous  topic; 

•  Automated  classification  with  large  numbers  of  prediction  variables  and  small 
numbers  of  cases,  where  software  has  been  implemented  and  tested  (Ramsey); 

•  Further  refinement  of  data  polishing  problems  for  messy  data  (Teng,  Teng, 
Benferat  and  Teng),  which  is  now  being  unified  with  Ramsey’s  automated 
classification  procedure; 

•  Algorithms  for  identifying  causal  structure  in  time  series  data  (Chu  and 
Glymour); 

•  Algorithms  for  identifying  when  unobserved  causes  are  acting  on  a  system  and  for 
identifying  the  causal  relations  among  those  unobserved  factors,  which  is  an 
important  expansion  of  data  analytic  methods  for  signal  source  detection; 

•  Development,  implementation,  and  testing  of  an  algorithm  for  identifying 
command,  control  and  reporting  relations  from  time  stamped  communication 
intercepts,  even  in  an  unknown  language  or  in  a  known  language  with  natural 
language  codes. 

Our  work  on  various  aspects  of  automated  causal  inference  will  be  the  subject  of  a 
weeklong  workshop  for  the  Department  of  Defense  in  August  of  2005.  The  work 
supported  by  ONR  has  also  resulted  in  the  preparation,  submission,  and  where  noted,  the 
following  publications/manuscripts : 

Danks,  David.  (In  press).  "Theory  Unification  and  Graphical  Models  in  Human 
Categorization."  In  A.  Gopnik  &  L.  Schulz,  eds.  Causal  Learning:  Psychology, 
Philosophy,  and  Computation.  Oxford:  Oxford  University  Press. 

Danks,  David,  &  Samantha  Schwartz.  (In  press).  "Causal  Learning  from  Biased 
Sequences."  In  Proceedings  of  the  27th  Annual  Meeting  of  the  Cognitive  Science 
Society. 

Scheines,  Richard,  Matt  Easterday,  and  David  Danks.  (In  press).  "Teaching  the 
Normative  Theory  of  Causal  Reasoning."  In  A.  Gopnik  &  L.  Schulz,  eds.  Causal 
Learning:  Psychology,  Philosophy,  and  Computation_.  Oxford:  Oxford  University  Press. 

Benferhat,  Salem  and  Choh  Man  Teng,  editors.  Special  issue  on  uncertain  reasoning, 
Parts  I  and  II,  International  Journal  of  Intelligent  Systems,  19(8-9),  2004. 

Teng,  Choh  Man.  Polishing  blemishes:  Issues  in  data  correction.  IEEE  Intelligent 
Systems,  19(2),  pages  34—39, 2004. 


Kyburg,  Henry  E.  Jr.  and  Choh  Man  Teng.  Randomization  and  Uncertain  Inference.  In 
Proceedings  of  Information  Processing  and  Management  of  Uncertainty  in  Knowledge- 
Based  Systems,  pages  1 177--1 184,  2004. 

Teng,  Choh  Man.  Coping  with  partially  corrupted  data.  Proceedings  of  the  International 
Conference  on  Machine  Learning  and  Applications,  pages  429—435,  2004. 

Glymour,  C.,  Social  Effects.  In  A.  Gopnik  &  L.  Schulz,  eds.  Causal  Learning: 
Psychology,  Philosophy,  and  Computation.  Oxford:  Oxford  University  Press. 

Glymour,  C.  Learning  the  Causal  Structure  of  Deterministic  Systems.  In  A.  Gopnik  &  L. 
Schulz,  eds.  Causal  Learning:  Psychology,  Philosophy,  and  Computation.  Oxford: 
Oxford  University  Press. 

Silva,  R.,  R.  Schemes,  and  C.  Glymour,  Automatic  Discovery  of  Linear  Latent  Variable 
Models,  Journal  of  Machine  Learning  Research  (submitted  for  publication). 

Gerdes,  D.  and  C.  Glymour,  Automated  Discovery  of  Command  and  Control  Structures 
from  Time  Series  of  Communication  Logs,  available  only  on  request. 

Chu,  T.  and  C.  Glymour,  Semi-Parametric  Causal  Inference  for  Time  Series,  Knowledge 
Discovery  and  Data  Mining  Conference,  2005. 

Englehardt,  F.,  C.  Glymour  and  R.  Schemes,  On  the  Number  of  Experiments  Necessary 
and  Sufficient  to  Determine  the  Causal  Structure  of  a  System,  Uncertainty  in  Artificial 
Intelligence  (submitted). 

J.  Ramsey,  Markov  Blanket  Fan  Search,  Technical  Report,  Laboratory  for  Symbolic 
Computation,  Carnegie  Mellon  University,  2005. 

Proposed  Research 

A  major  focus  area  for  FY  05  will  be  on  how  extend  our  work  on  the  use  of  data 
associated  with  complex  systems,  specifically  to  merge  the  methods  we  have  developed 
with  experimental  interventions  so  as  to  minimize  the  number  of  experimental 
interventions  required  to  determine  the  structure  of  the  system.  These  results  are  of 
practical  value,  but  they  would  be  of  more  use  if  the  methods  could  be  extended  in  three 
ways:  (1)  to  deterministic  systems;  (2)  to  systems  that  may  have  unrecorded  features  that 
influence  multiple  recorded  features  (e.g.,  external  heat  source  influences  on  space  or 
aircraft;  unrecorded  signals  and  sources  coordinating  enemy  movements,  etc.),  and  to 
feedback  systems.  Our  continuing  work  addresses  all  three  problems.  In  particular,  we 
have  in  hand  a  modification  of  search  algorithms  for  deterministic  systems,  and  are  now 
testing  it,  and  we  have  reason  to  believe  that  when  there  may  be  latent  unrecorded 
common  causes  of  recorded  variables,  all  causal  connections  among  recorded  variables 
can  nonetheless  be  identified  in  21og2(N)  +  1  multiple  intervention  experiments. 


SUMMARY 


Very  substantial  progress  has  been  made  in  all  three  areas  of  research  undertaken  in  the 
FY  04  period  of  performance  for  this  effort  in  Human  Systems  Technology.  A  follow-  on 
year  has  been  funded  to  continue  this  promising  work.  During  FY  05  work  on  haptic 
information  displays  will  continue,  as  will  that  addressing  new  methods  for  Knowledge 
Discovery  and  Data  Mining.  Additionally,  fundamental  work  will  be  initiated  on  (1) 
trustworthy  software  agents,  realized  through  a  more  adequate  theory  and  implementation 
for  adjustable  autonomy;  and  (2)  exoskeletons  for  human  performance  enhancement. 


